An integrated expert weight determination method for design concept evaluation

Expert weight determination is a critical issue in the design concept evaluation process, especially for complex products. However, this phase is often ignored by most decision makers. For the evaluation of complex product design concepts, experts are selected by clusters with different backgrounds. This work proposes a novel integrated two-layer method to determine expert weight under these circumstances. In the first layer, a hybrid model integrated by the entropy weight model and the Multiplicative analytical hierarchy process method is presented. In the second layer, a minimized variance model is applied to reach a consensus. Then the final expert weight is determined by the results of both layers. A real-life example of cruise ship cabin design evaluation is implemented to demonstrate the proposed expert weight determination method. To analyze the feasibility of the proposed method, weight determination with and without using experts is compared. The result shows the expert weight determination method is an effective approach to improve the accuracy of design concept evaluation.

www.nature.com/scientificreports/ a specific application area of MADM methods 1 , and the most frequently cited studies in this area are listed in Table 1. Although researchers have made efforts to optimize the method to obtain criteria weights, expert weight is rarely mentioned. In Fig. 2, from the cognitive system of design concept evaluations, we can infer that an expert needs to make an individual judgement based on their cognition and preference. Thus, the judgement is subjective, depending on the expert's knowledge, culture, experience and aesthetic. However, it is practically impossible to form a group of experts with a similar background. The drawback of an individual in the expert group may influence the accuracy of experts' judgement.
Once the weight of each expert is distributed equally or ignored, the final decision may lead to an incorrect result. Thus, some researchers started to develop an appropriate method to determine the expert weight. In their investigations, not every selected expert is familiar with all the criteria requiring judgement, and some specialized criteria may be beyond the cognition of some experts. Thus, the contribution of experts may not be equal   www.nature.com/scientificreports/ in the decision process. It is essential to determine the weight of experts in the decision process to eliminate any deviation caused by the experts' imprecise cognition. Generally, there are three ways to solve weight determination problems: the subjective approach, the objective approach and the integrated approach, as shown in Fig. 3. For the subjective methods, the weight of experts is calculated by integrating the evaluation of each other expert, depending on their age, attitude, experience etc. 21 . The objective methods are based on available evaluation data, and no extra information is required. Hence, the expert weight is computed based on an appropriate mathematical model 22 . Compared to the subjective methods, they are generally more objective, but they ignore the expert's personal preference. In this part, the expert weight determination methods are reviewed.
In the early stage, subjective methods dominated in studies on expert weight determination. Expert weights were assigned by the supervisors or by comparison between individual expert groups. In the subjective expert weight determination, pairwise comparison matrices were established, then methods with a geometric mean were proposed to determine the expert weight. Multiplicative AHP 23 and Simple Multi-Attribute Rating Technique (SMART) are two key approaches in subjective expert weight determination 24 .
Ramanathan and Ganesh 23 were the first scholars to develop the Multiplicative AHP framework in the expert weight determination area. They proposed an eigenvector based method by comparing the experts' influence in pairs, as interpersonal comparison can help experts achieve a consensus without any interaction 23 . Barzilai et al. 24 improved the Multiplicative AHP method and explained the relationship between the Multiplicative AHP method and the SMART method. The geometric mean of the scaled gradation indices or values is applied in both methods 21 . As well as the methods mentioned above, Delphi is another competent subjective approach in expert weight determination. Azadfallah 25 applied the Delphi technique by comparing the attributes by experts in pairs, then computed the weights with the eigenvector method.
Subjective expert weight determination methods are based on relative judgements. To make precise judgement, the experts are expected to be familiar with every attribute. However, in real-life projects, it is not practical for every expert to meet all the requirements. Moreover, once the number of attributes, alternatives, or experts increases, pairwise comparative work would be a huge project. Due to the heavy workload, pairwise comparative judgements are not easy to implement in complex MADM problems. Objective weight determination methods have developed rapidly in recent years. There are three objective expert weight determination methods, as well as some special methods such as the Markov Chain's theory 26 .
In similarity-based methods, expert weights are determined by distinguishing the expert's evaluation or measuring the expert's distance to the aggregated decision 27,28 . The similarity-based study is very close to the TOPSIS method, in which the expert whose decision has minimum distance to the ideal solution has the highest weight. Yue 28,29 proposed a modified TOPSIS method, to determine the expert weights by measuring the distance between the expert judgement and the ideal decision. Yang proposed a rough group decision matrix to determine the expert weights 30 . Wan 31 constructed a bi-objective program which could maximize the minimum degree of acceptance and minimize the maximal degree of rejection, and also introduced three approaches to measure the distance between the individual preference and the group. Jiang 32 introduced a novel method to measure the distance between the expert preference and the ideal solution.
Index-based methods are divided into two groups: consensus-based methods and consistency-based methods. In consensus-based methods, experts are assigned to adjust their preferences, or vary their weights to make their judgements more similar 33 . Pang 34 developed a non-linear programming model, and determined the expert weight by maximizing the group consensus where the expert weight adjusts adaptively with the experts' decision. Xu 35 also used a non-linear model, and proposed a genetic algorithm in expert weight determination. Dong 36 introduced a consensus reaching process in group decision making. The method is proposed for noncooperative expert groups, and the expert weights are determined dynamically by a self-management mechanism. Consistency-based methods are superior in inaccurate judgements, using consistency indexes against the group decision to determine the weights of experts 37 . When some experts have higher reliabilities compared to others, the decision makers need to reach a consistent view through negotiations. Liu 38 introduced an expert weight adjusting method based on the consistency-based method. The expert weights are firstly computed by the AHP method and the entropy method, and then optimized by the consistency level of a black-start result. Chen 39 established a collective consistency matrix of all experts and determined the weight of experts by the consistency degree of each expert. www.nature.com/scientificreports/ When the group is large, a clustering-based method may be appropriate in real-life problems. A multi-level weight determination method may optimize weight via different models according to the features of the current layer 40 . Sometimes experts are assembled in several specific groups, and the experts in the same group have a similar background, but the gaps between groups are large. As shown in Fig. 4 by Liu 40 , a two-layer model is proposed based on the 2-tuple linguistic (2TL) model. In the first layer, decision makers are separated into clusters by certain criteria. The weight of the group depends on the importance of itself. The clusters are regarded as small systems with similar individual information, which is a well-organized system. Liu utilized an Entropy Weight Model to reach consistency. In each cluster, the experts' status, occupations and experiences are close. Therefore, their judgements should be similar but not identical 41 . To reach a consensus, a Minimized Variance Model is implemented in this layer to seek a minimized deviation among all variables.
The literature proposed problem-solving methods through a specific mathematical model or operator to determine expert weights, through methods such as comparing the deviation between the expert preference and the ideal solution, or simply adjusting the expert weight to make the experts achieve consensus. However, there are still some drawbacks. First, as some of the mathematical models or operators may focus on the overall average, some information that deviates from the average level may be ignored. Second, existing studies usually obtain the expert weight by a specific method. Once the problem is complex, one single method cannot reflect a real-life problem accurately.
The integrated method relies on an integration of two or more methods to eliminate the drawbacks of a single approach or simplify a complex method. As integrated methods show their advantage in solving complex GDM problems which have become more pronounced in recent years, an increasing number of integrated expert weight determination methods are presented in decision making 22 . Qi 42 proposed models based on various conditions under interval-valued intuitive fuzzy decision environments and determined both criteria weight and expert weight. First, they introduced a method to measure the gap between decision matrices and the ideal decision matrix, and then they developed an approach to evaluate the similarity degree between individual decision matrices. Liu 43 proposed another expert weight determination method. The method integrated both subjective and objective expert weight determinations in decision making. First, a plant growth simulation algorithm is applied to get the generalized Fermat-Torricelli point of individual preferences with interval number decision matrices. Then a similarity-based expert weight determination method is used. Finally, the expert weights obtained from both methods are aggregated. Jabeur 44 also determined the expert weight based on subjective and objective components.
In product design concept evaluation, a group of experts must be selected in a proper way to ensure the correctness of the assessment. In complex product design concept evaluation projects, as shown in Fig. 5, experts are normally selected from experienced consumers and expert producers 45 . www.nature.com/scientificreports/ Normally, most expert producers are designers and manufacturers. Hence, we can categorize the decision makers into the designer cluster (DC), the manufacturer cluster (MC), and the consumer cluster (CC). The cluster category can perfectly reflect the clustering information of experts. In addition, the integration of subjective and objective weights can improve the accuracy of the design concept evaluation 46 . However, recent studies seldom consider the subjective expert weight due to the workload when the expert group is large.
In our work, we integrated the subjective and objective expert weights with a 2-layer cluster weight determination. The distribution of expert preferences in the clustering-based method is illustrated in Fig. 6. In the cluster layer (Layer 1), the Shannon entropy model can illustrate the organization of the condition of the clusters, but cannot well reflect the individual expert preferences. Hence, an aggregated method integrated by AHP and entropy weight model is proposed under this layer. In the subsection layer (Layer 2), the decision makers in the same cluster are the experts with similar knowledge and background, so their preferences should be highly consistent 40 . Under the circumstances, subjective pairwise comparison is omitted here, because on the one hand, it may largely increase workload, and on the other hand, the influence may not be obvious. Thus, an objective minimized variance model is used here.
The rest of this paper is organized as follows. In "Preliminaries", relevant expert weight determination methods are reviewed. In "Methodology", the conceptual framework of expert weight determination is presented. In "Case study", a real-life example is implemented and related analysis is presented. In "Comparison of methods with and without using weight determination", a comparison with and without using expert weight determination is discussed. In "Conclusions", the conclusion is provided.

Preliminaries
This study proposes a novel integrated expert weight determination method. Before presenting our method, some related expert weight determination methods are reviewed.
The multiplicative AHP method. Multiplicative AHP is a significant subjective expert weight determination method. The method is easy to implement in real-life cases 23 . Similar to other subjective methods, the Multiplicative AHP determines the expert weight through pairwise comparisons. Initially, each expert in the decision maker group is allocated to assess every group member. It may lead to a personal upward bias 21,26 . To eliminate the error caused by the expert's assessment, in Honert's study, each expert's comparisons with the expert him/ herself are no longer counted. For example, each expert in the decision maker group with G experts only needs to make (G − 1) comparisons.
The expert weight determination method of Multiplicative AHP in Honert's approach can be summarized in the following steps 21 .
Step 1: Assume the expert group has G members, DM y is a member of the expert group. As is shown in Table 2, every expert is assigned to make a linguistic comparison between attributes by individual judgement using pre-provided words (Very strong preference / Strong preference / Definite preference / Weak preference / Indifferent). S k and S j represent the expert's preference for the alternatives A k and A j , respectively. Hence, comparison between S k and S j can be converted into a numerical value δ jky by a geometric scale based on the content of Table 2. Then matrix {r jkd } can be obtained by the equation below: where γ represents a scale parameter, with a frequently-used value of ln2 . Next, approximate vector p of stimulus values can be determined by the logarithmic least-squares method by Lootsma 47   www.nature.com/scientificreports/ Step 2: Substituting w j = lnp j ,w k = lnp k and q jky = lnr jky = γδ jky , the function transforms to The associated set of normal equations can be transferred by w j to where j = 1, 2, . . . , G . The variate j has the same value range in the following equations in this part.N jk denotes the cardinality of S jk . According to Lootsma, N jk = G − 2 . (G − 1) comparisons are made for each expert, the maximum pairwise comparison is (G − 1)(G − 2)/2 , and Eq. (4) can be rewritten as: The equation can be simplified as: Step 3: For any alternative A k and A j , from Table 2, we have q jky = −q kjy , S jj is empty and q jjy = 0.
Assume each expert made all the comparisons, then G k=1,k� =j w k = 0 , the equation can be written in the following form: Thus the expert weight of decision makers p j can be computed by Eq. (8).
After normalization with the equation below, the subjective expert weight w s i can be determined.

2-tuple linguistic and the model of the two-layer weight determination method.
Definitions. As described in "Introduction", the two-layer weight determination method is a critical cluster-based method. The 2-tuple linguistic (2TL) provides the environment of the two-layer method, and this part introduces the rationale to select 2TL. Due to the uncertain environment, fuzzy set 48 , rough set 49 , grey decision 50 , and some other extended methods are applied in MADM problems 51 . 2TL is a model based on the linguistic fuzzy set proposed by Herrera and Martínez 52 . www.nature.com/scientificreports/ fuzzy set to describe the flexibility of words. Reviews and extended models of 2TL can be found in Martínez 53,54 and Malhotra 55 . Xu proposed non-linear aggregation operators in decision making with uncertain linguistic information 56,57 . Wang and Hao 58 introduced a new 2TL model based on 'symbolic proportion' to preserve the integrity of information, as the proportional 2-tuples can well illustrate the uncertainty of the linguistic judgement. As linguistic term sets distribute uniformly and unsymmetrically, Herrera 59 defined the term sets as unbalanced linguistic term sets to demonstrate the non-linear thinking of human beings, and proposed a method integrated by a representative algorithm and a computational approach. As an effective method in MADM, the 2TL model has produced crucial results in various areas such as quality assessment 60 , web system design 61 and group decision making 62 . The 2TL model perfectly matches most subjective problems as the linguistic variables have their advantage in expressing 'approximate information' . For example, we can use the 2TL model to describe 'how young the person is' , using the linguistic words 'very young, young or not young' .
In design concept evaluation, experts are assigned to evaluate plans on a large number of different attributes. Some of the attributes do not have an exact value, and thus the evaluation of the experts may be subjective and imprecise. Approaches such as the House of Quality method with multi-point scale measurement and intuitive fuzzy set are proposed to tackle the challenge. However, the 2TL is suitable for design concept evaluation problems for two reasons. First, design concept evaluation is complex multi-attribute decision making based on the aggregation of the decision makers' judgements, which belongs to the MADM problem. Second, experts prefer to make judgements in natural language with some adverbs of degree such as 'very' , 'extremely' etc. 55 In design concept evaluation, raw data and information may be uncertain, imprecise and vague, which well matches the category of 2TL model problem-solving, and the evaluation data can be used as raw data in some MADM methods such as TOPSIS and VIKOR.
Computing process. The computing process of the two-layer determination method is described as: Assume S = {s i |i = 0, 1, 2, . . . , u} is an LTS. The criteria (attribute) group and the expert group are expressed as C = {c 1 , c 2 , . . . , c n } and E = {e 1 , e 2 , . . . , e m } respectively. A = {A 1 , . . . , A l } represented as l alternatives. The experts (E) are assigned to evaluate the alternatives (A) according to the criteria (C) using the LTS (S). The experts are required to give their evaluation using linguistic terms. As for expert e k , the decision matrix X k is described as: where i, j and k represent the index of alterative ( A i ), criteria ( c j ) and expert ( e k ) respectively. In the two-layer method, m experts can be divided into f clusters, expressed as G = {G 1 , G 2 , . . . , G f } . There are m y experts in the cluster G y , where f y=1 m y = m and y ≤ f .
Step 1: Minimized variance model in layer 2. The minimized variance model relies on minimizing the total variance preference in a cluster. The matrix X k can be transferred into 2-tuple B k as follows: where b k ij is a 2-tuple. In the second layer, assume expert e k is the p th expert in the cluster G y , we denote by B yp the p th decision matrix in G y . Hence, we can rewrite Eq. (20) as: where p ≤ m y . To make the computing process simple, the matrix of an expert is converted to a vector: where t = l × n . The decision matrix of cluster G y is To get the optimized solution, it is required to make the sum of the variance of the attributes weight small.  1 , α 1   The summary of variances is: The optimization model is shown below: Then, the optimal weight value of variate y can be determined as * y = ( 1 , 2 , . . . , m y ).
Step 2: Entropy weight model in Layer 1. In the first layer, the TAM operator is applied in the entropy weight model. As the basic equation of entropy, we have: where the constant k is calculated as k = 1/lnn . Let D = (d yj ) f ×n as the decision matrix of the clusters. The equation above can be converted to:  www.nature.com/scientificreports/ Let w y = (w 1 , w 2 , . . . , w f ) T be the weight vector of clusters. We can compute the cluster weight by the equation below: Step 3: Expert weight calculation. By now, the expert weight in cluster * yi and the cluster weight w y are determined, and the i th expert weight in cluster G y , the expert weight w e can be obtained by multiplying the two weights.
Thus, the expert weight can be determined by the two-layer weight determination method.

Methodology
In design concept evaluation, attributes are selected from multiple dimensions, such as shape, color, ergonomics, material and manufacturing technology, etc. The experts are categorized into clusters depending on their different backgrounds. Therefore, a two-layer expert weight determination is a good solution to design concept evaluation in NPD. However, there are still some drawbacks to this method. The complexity of different products varies, experts in the customer cluster may not be familiar with how the product works, and they may make judgements relying only on their experiences. Moreover, if the experts' weights merely depend on the objective method, the subjective preferences of the experts are ignored. Thus, an integrated two-layer expert weight determination approach is proposed, as shown in Fig. 7. In the first layer, we used a mixed weight determination method based on the AHP and entropy weight model. In the second layer, a minimized variance model as mentioned above was applied. After that, the final expert weight is determined by combining the two layers' results. The proposed method is illustrated below.
Step 1: In this step, a 9 LTS S is constructed according to the 2TL environment. The experts are assigned to give their evaluation using linguistic terms. As for expert e k , the decision matrix A k is described as: where i, j and k represent the index of alternative ( A i ), criteria ( c j ) and expert ( e k ), respectively. The matrix A k can also be shown with 2-tuple: where b k ij is a 2-tuple. In the second layer, assume expert e k is the p th expert in the cluster G y , we denote by B yp the p th decision matrix in G y . Hence, we can rewrite Eq. (33) as: where p ≤ m y . To make the computing process simple, the matrix of an expert is converted to a vector: where t = l × n . The decision matrix of the designer cluster is: Thus, the decision matrix of layer 2 can be represented as F = (F 1 , F 2 , . . . , F f ) T .
Step 2: The weight of experts in the corresponding cluster is calculated by the minimized variate model in the second layer.    www.nature.com/scientificreports/ Generally, experts in the same cluster have a similar background, which means the experts have a high possibility of having similar preferences for the alternatives. It is required to minimize the variance of the attributes of the clusters. To get the optimized solution, it is required to make the sum of the variance of the attributes weight small. Using Eqs. (37) to (40), we can get the weight evaluation of the designer cluster: where F y j is the arithmetic mean of F y ij , and yi = ( 1 , 2 , . . . , x ) T indicates the weight vectors of experts in cluster G y , and the summary of weight vectors is 1. The definition of symbol * in this equation is: The summary of variances is: The optimization model is shown below: Then, the optimal weight value of variate y can be determined as * y = ( 1 , 2 , . . . , x ) . Similarly, the optimal weight value of the rest of the clusters in the second layer can be determined in the same way.
Step 3: Next, we determined the cluster weight in the first layer by a hybrid method. In this step, we use the entropy weight model to determine the objective weight of clusters.
(37)  Table 3. Integer-valued cluster important judgement designating the gradations made by decision makers.

Comparative judgement Gradation index
Very strong importance for G α versus G β − 8 Strong importance for G α versus G β − 6 Definite importance for G α versus G β − 4 Weak importance for G α versus G β − 2 Indifference between G α versus G β 0 Weak importance for G β versus G α + 2 Definite importance for G β versus G α + 4 Strong importance for G β versus G α + 6 Very strong importance G β versus G α + 8 Table 4. Form of the data recorded in the table based on pairwise comparative judgements. www.nature.com/scientificreports/ In the first layer, the TAM operator is applied in the entropy weight model. For the cluster G y , as the basic equation of entropy, we have: where the constant k is calculated as k = 1/lnn. Let D = (d Dj ) f ×n as the decision matrix of clusters. The equation above can be converted to:

the weight vector of clusters. Then
The result of w o can be computed as the weight vector of clusters by the objective method.
Step 4: Here in this step, a subjective method is also implemented in cluster weight determination. The experts are assigned to assess the importance of three groups by the AHP method. They give their comparison of preferences among the clusters.
First, the experts are assigned to compare the clusters, the comparison among G 1 , G 2 , . . . , G f . Every expert is assigned to make a linguistic comparison between clusters by individual judgement using specified words (Very strong importance / Strong importance / Definite importance / Weak importance / Indifferent). The comparisons made by experts in G y between cluster G α and G β are converted into a gradation index by geometric scale shown in Table 3. The comparison can be recorded in the comparative table in the form of Table 3. Where G x y (G α /G β ) represents the comparison of G α /G β made by the x th expert in cluster G y . After that, the arithmetic means of the comparison by cluster can be computed, which is also shown in Table 4. δ αβy denotes the arithmetic mean of the comparison G α /G β made by experts in cluster G y , then we have δ αβy = G y (G α /G β ).
The comparison matrix of clusters {r αβy } can be determined by the equation below: where γ represents a scale parameter, with a frequently-used value of ln2 . Next, the approximate vector p of stimulus values can be determined by the logarithmic least-squares method by Lootsma 47 . The vector p minimizes where p α and p β represent the relative power of G α and G β made by G y , respectively. Assume q αβy = lnr αβy and w α = lnp α , expression (45) can be described as: The associated set of normal equations can be transferred by w α to: where α = 1, 2, . . . , f , and the variate α has the same value range in the following equations in this part.N αβ denotes the cardinality of G . According to Lootsma, we have N αβ = f − 2 . (f − 1) comparison made for each expert, the maximum pairwise comparisons are (f − 1)(f − 2)/2 , and Eq. (47) can be rewritten as: The equation can be simplified as: For any cluster G α and G β , from Table 2, we have q αβy = −q βαy , G αα is empty and q ααy = 0. Thus the expert weight of decision makers p α can be computed by Eq. (48).
After normalization with the equation below, the subjective expert weight w s i can be determined.    www.nature.com/scientificreports/ Step 5: In Step 3 and Step 4 above, the objective cluster weight and the subjective cluster weights were determined. Here in this step, combined weights are determined using the equation below to get the combined weight of clusters.
where µ is the adjusting coefficient, here µ ∈ [0, 1] represents the superiority of the subjective method over the objective method in the combination. When µ > 0.5 , the subjective determination of the DM group is superior, on the contrary, when µ < 0.5 , the objective method is superior, and when µ = 0.5 , both methods made the same contribution in the cluster weight determination.
Step 6: Thus, the final weight of each expert ω ui can be computed by: Ethical approval. This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent.
No informed consent was required, because the data are anonymized.

Case study
In our study, the new two-layer expert weight determination method is applied in optimization of the cabin design plan for a mid-sized cruise ship. Before the evaluation, three design schemes are proposed, and a decision needs to be made from the three alternatives. To make the decision, a 10 attribute criteria index shown in Table 5 is fixed to determine the alternatives. A 30-expert group is formed with 10 cruise ship interior designers, 10 cruise ship manufacturing specialists and 10 customers with over two cruise trip experiences. The raw data is shown in Appendix 1.
First, according to Eqs. (32) to (40), a minimized variance mode is established. Then the optimized lambdas(λ) are determined by the software Lingo. The corresponding weight of experts in each group are shown in Table 6.
After that, according to the TAM operator from Definition 3, the decision matrix D is shown in Table 7. We can calculate the objective weight of the clusters w o = (0.424, 0.373, 0.203).
We get the pairwise judgement about the importance of the clusters from Appendix 2. The data in Table 8 is the arithmetic mean calculated from subjective pairwise judgements using the weighting method in each cluster. Here in the table, we leave the cell blank when the cluster is compared against itself. For other cells, the symbol "-" means that the corresponding row or column is not permitted to be compared with others. The normal equation can be expressed as: In the equation, q αβy = γδ αβy , Eq. (55) can be simplified as: For DC, α = 1 , we have The weight of the clusters can be calculated by Eqs. (50) to (52). In this case, the subjective weights of clusters are: The integrated weight of the clusters can be computed by Eq. (53), and when µ = 0.5 , the weights of clusters are: The final weights of each expert w are shown in Table 9.

Comparison of methods with and without using weight determination
Among MADM methods, TOPSIS is widely used in supply chain management, design concept evaluation, business and marketing management, and some other fields. To illustrate the influence of the expert weight, a comparative analysis is made using the TOPSIS method with and without expert weight determination.
The steps of the TOPSIS method are shown here.   www.nature.com/scientificreports/ Step 1: Normalize the decision matrix.
For a decision matrix X = x ij has m alternatives and n criteria,1 ≤ i ≤ m and 1 ≤ j ≤ n , respectively.r ij represents the normalized matrix.
Step 2: Compute the matrix with criteria weight.
where w j represents the weight of criterion j.

Step 3: Calculate the positive ideal solutions (PIS)/negative ideal solutions (NIS).
where v + j = {max v ij for benefit − type criterion; min v ij for cost − type criterion} and v − j = {min v ij for benefit − type criterion; max v ij for cost − type criterion}.
Step 4: Compute the separation measures of alternatives. Let S + i / S − i be the distance between the alternative and the PIS/NIS.

Step 5: Calculate the closeness index (CI) value.
The rank of the alternatives can be determined by comparing CIs . The alternative with the highest CI is the best solution.
The criteria weights are determined by the subjective method, shown in Table 10.
According to the TOPSIS method, the relative variates are calculated. S + i ,S − i , CIs of alternatives with and without expert weights are shown in Table 11.
From Table 11 and Fig. 8, it is clear that the ranks of alternatives are different with and without considering expert weight. In Fig. 8, both methods demonstrate alternative A 1 is the least ideal option, and far inferior to the other two solutions. However, when the expert weight is considered, the most feasible alternative changed from A 3 to A 2 . On the other hand, the variation of S + i ,S − i , CI with and without expert weight is obvious from Table 11. The difference is caused by the weight of experts. We can infer from the comparative analysis that if the gap between alternatives is huge, the expert weight determination may not influence the final decision. Otherwise, if a gap between alternatives exists but is not particularly large, the expert weight determination may be considered more in decision making. Ignoring the expert weight may cause us to miss the best alternative in real-life projects as verified in this section.

Conclusions
Expert weight determination in design concept evaluation is a critical part that is usually ignored by decision makers. A proper weight determination can make the decision making process more accurate. This paper presented an integrated two-layer expert weight determination method under a complex design concept evaluation process. In some complex problems, experts are divided into clusters by certain characteristics, and the weight of experts can be calculated by individuals (layer 1) and clusters (layer 2). In the first layer, the minimized variance model is presented to determine the individual weight in each group. In the second layer, a hybrid weight determination method is proposed by combining the entropy weight method and the AHP method. A case study in cruise ship cabin design was implemented using the proposed method. Comparison of the results showed that weight determination in a complex product design process is essential, and may sometimes cause different outcomes. (59)